Ok, I am doing quite a bit of research on Apache lately (for my project), and I am hoping some of you can help me out by answering a question or two that I have... or at least pointing me in the right direction (as long as it's not something general that I've already looked at, like apache.org website)
I am wondering how (if at all) does Apache handle moving files from disk to memory (caching, so to speak)?
Is it mainly through mod_cache? mod_disk_cache? mod_mem_cache?
I've looked at those, and it seems to me that I can only change a few settings within these modules (max/min cached object size, etc). Is there a lot more int the actual source code that I could look at and optimize?
Or not at all?
I am asking because the topic of my research concentrates on improving the delivery of multimedia files through Apache... and one of the aspects I am looking into right now is simply caching all the files from the disk to RAM/memory to cut down on the time it takes for the file to become available to client (or apache server, in the first place).
Thanks in advance....
I can only give you some hints as I don't know too much about the apache source code.
Apache can use the unix systemcall "readfile" for static files. This speeds up the transfer from disk to memory significantly. But it is not available on Windows eg.
Next, you have got your operating system's filesystem cache.
For *BSD and Linux systems, all free memory will be used to cache files from the filesystem. If the file has been read once and the cache has not been deleted yet, it will be served in no time.
For caching dynamic sites (php, shtml, perl), you need a caching module, eg. mod_cache.
But many php programs are implementing caching on their own and achieve a much more sophisticated solution caching only parts of a page. An external module could not know about the internal structure of the program and could only cache the full page at once.
You are on the wrong track if you want to speedup loading with mod_cache. You need to tweak your operating system cache settings instead.
Apache is available in source code, but there is probably few to nothing you can optimize. Download it from apache.org (http://httpd.apache.org/) and see yourself.
and one of the aspects I am looking into right now is simply caching all the files from the disk to RAM/memory to cut down on the time it takes for the file to become available to client (or apache server, in the first place).
As told above, after they have been loaded, they are in your filesystem cache. Load all media once and use enough free ram, you got everything cached.
In reality, in a production environment, it doesn't work that simple though ;)
You may want to use a RAM disk and copy all contents there if you really think (or know) that your hard drives are the bottleneck.
M.
So no matter what I try to do to Apache and its modules, I can't improve the performance by anything noticeable...?
I just wanted some pointers in where to go from there... since I am being pushed into actually editing the code of Apache and its modules, but I am not finding any sensible reasons into doing so.
ok, another question...
here is what I'm trying to accomplish:
1. Have Apache serve an HTML file to the client
2. Have Apache scan through the HTML to see what images/multimedia objects it references
3. Based on those scans, have Apache load objects from disk cache into memory cache.
4. Several milliseconds later, when the client requests an image, Apache serves it from the memory cache saving us microseconds in disk access time.
Is there any way to pull that off in Apache, through modules such as mod_mem_cache or mod_disk_cache (the experimental ones)... or without modifying the modules directly but specifying the parameters through the Apache config file itself or something?
Help!
Really not sure if I am the right person to talk to about such internal details. Maybe check the apache mailing lists.
But from my view, it would not help anything.
You get the same result. File is loaded from disk once and cached in memory. But you are adding the overhead of parsing the file. And the browser client is probably much faster rendering the page if your server is under load. So it will request the files before you cached them anyways.
If you need a faster server, remove modules and features from apache and tune the MPM settings. Tune your OS cache too.
Maybe Apache isnt for you if it isnt fast enough. There is faster servers which trade speed to features.
I dont know anything about apache caching modules other than mod_proxy which serves a different purpose. But maybe a Proxy Server is a solution to your problem?
M.
Proxy server? Maybe... can it do something along the lines of what I'm trying to accomplish.
Remember, this is a research I've been given and asked to do, so it's not like I think this is the best way to get this done...
And where in the world is apache mailing list? Too bad they don't have some sort of apache forum (or at least an active sub-forum dedicated to apache on some website)...
Proxy server? Maybe... can it do something along the lines of what I'm trying to accomplish.
Remember, this is a research I've been given and asked to do, so it's not like I think this is the best way to get this done...
And where in the world is apache mailing list? Too bad they don't have some sort of apache forum (or at least an active sub-forum dedicated to apache on some website)...
Umm. Sorry, I am finished with you here.
Who gave you this research? Who made you think that you could lead this research to success?
- You dont know what a proxy is
- You think theres a way to make apache faster by your work alone than the thousands of volunteers working on the apache project
- You cant find the apache maling list....
Over and out.
Good night, boy. Have fun with your ´research´...
M.
Umm. Sorry, I am finished with you here.
Who gave you this research? Who made you think that you could lead this research to success?
- You dont know what a proxy is
- You think theres a way to make apache faster by your work alone than the thousands of volunteers working on the apache project
- You cant find the apache maling list....
Over and out.
Good night, boy. Have fun with your ´research´...
M.
No problem... thanks for 'helping out'.
As for the research goes, I had no clue if I could get this research to success or not because it's not something I thought of but something I was passed on by a professor. That is why I am doing all the 'research', to figure out what's the best possible way to approach this and is HIS idea even possible at all.
And no, I don't think I can do anything even close to the work all the developers on Apache project are doing... but it would be great to know if it's something that's beeing looked at at the moment, by someone, somewhere...
And as far as the mailing list goes, I didn't even know there is one (sorry, but not all of us have been gifted with all the information as you have)... I just found out there's a yahoo group as well, so I'll ask for more help in there.
As far as you go, there really is no need to look down on people due to their lack of knowledge in a certain area, or being less informed than others... not all of us have a chance to sit at the computer 24/7, browse the internet and absorb all this information like some of you, ok? But whatever, it's your life, your mentality... you go through your own life any way you feel like. I don't care.
A few pointers here and there would've been helpful though... I didn't ask you to write a book or anything.
As far as you go, there really is no need to look down on people due to their lack of knowledge in a certain area, or being less informed than others... not all of us have a chance to sit at the computer 24/7, browse the internet and absorb all this information like some of you, ok? But whatever, it's your life, your mentality... you go through your own life any way you feel like. I don't care.
A few pointers here and there would've been helpful though... I didn't ask you to write a book or anything.
I am really sorry if this appeared to you this way. It wasn`t meant to.
I don`t mean to look down on anybody. Just your specific problem seemed to me that a `professor` gave you this task and this professor doesnt know what he is seeking for yet. Just a ´make it better` plan.
I hope some things that I posted can help you to get your problem solved. If not, ... I am just a human too...
Anyways, this is just some frickin internet forum.... I don`t really care either... ;) Have a good night!